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CROSS-REFERENCE TO RELATED APPLICATION 

This is a continuation-in-part of co-pending U.S. 
application No.: 09/567,786, entitled "Multiple Sampling via 
a Time-indexed Method to Achieve Wide Dynamic Ranges/' filed 

10 on May 9, 2000 and by David Yang, et al . , one of which is a 
co-inventor hereof. This application is related to U.S. 
application No .: 0 9/823 , 838 , entitled "Method And Apparatus 
for Storing Image Information for Multiple Sampling 
Operations in a Digital Pixel Sensor," filed on March 30, 

15 2 001, which is hereby incorporated by reference in its 
entirety. 



BACKGROUND OF THE INVENTION 
1. Field Of The Invention 
2 0 The invention relates to image sensor systems, and in 

particular, the present invention relates to an image sensor 
with improved color reproduction capability. 



2 . Description of the Related Art 

25 Digital photography is one of the most exciting 

technologies that have emerged in the past years . With the 
appropriate hardware and software (and a little knowledge) , 
anyone can put the principles of digital photography to 
work. Digital cameras, for example, are on the cutting edge 

30 of digital photography. Recent product introductions, 

technological advancements, and price cuts, along with the 
emergence of email and the World Wide Web, have helped make 
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digital cameras the hottest new category of consumer 
electronics products. 

Most digital cameras use an image sensor or a 
photosensitive device, such as a charged-coupled device 
(CCD) or a Complementary Metal-Oxide Semiconductor (CMOS) 
device to sense a scene. The photosensitive device reacts 
to light reflected from the scene and can translate the 
strength of that reaction into electronic charging signals 
that are further digitized. Because the image is actually a 
collection of numeric data, the image data can easily be 
downloaded into a computer and manipulated for more artistic 
effects . 

Figure 1 illustrates an image sensor or photosensitive 
chip 100 which may be used in an image capturing device, 
such as a digital camera, for either stationary or video 
photography. Image sensor 100 produces digital image data 
representing the light intensity of the scene being 
captured. Image sensor 100 is typically fabricated on a 
single substrate, such as in a CMOS fabrication process, and 
includes a two-dimensional array of photosensitive devices, 
also referred to as photodetectors . The number of the 
photodetectors in image sensor 100 typically determines the 
resolution of digital images resulting therefrom. The 
horizontal resolution is a function of the number of 
photodetectors in a row 102, and the vertical resolution is 
a function of the number of photodetectors in a column 104. 

Each of the photodetectors in an image sensor comprises 
a photosensor that produces an electronic signal when it is 
exposed to light. Generally, the photosensor is a 
photodiode or a photogate in a CMOS sensor. Figure 2 
illustrates an exemplary structure of a photodetector where 
a photodiode 120 is modeled as a current source 122 and a 
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capacitor 124. When a reset signal is applied at a Reset 
terminal 13 0, the capacitor 124 is fully charged to nearly 
the Vcc voltage through an NMOS transistor 128. After 
capacitor 124 is fully charged, photodiode 120 is ready for 
5 light integration. 

When photodetector 12 0 is exposed to light, photons 
from incident light 126, filtered through a filter 136, 
impinge upon photodetector 120 and cause a change in the 
conductivity of the photodetector which is represented as a 
10 change in the current flow of current source 122. Current 
source 122 discharges capacitor 124 at a rate dependent on 
the number of photons striking photodetector 12 0. The 
voltage across capacitor 124 is thus dependent on the total 
number of photons striking photodetector 12 0. An output 
15 voltage signal V ou t generated at output node 129, which is 
the voltage across capacitor 124, is indicative of the 
integrated light intensity between the time that transistor 
128 is turned off and the time that light 126 incident on 
photodetector 12 0 is turned off or the time the readout 
20 process begins. An optional circuit 132 may be included to 
enhance the electronic signal V out to a desired level so that 
the output signal, i.e. the pixel charge signal, can be 
effectively coupled to the subsequent circuitry in the image 
sensor. The pixel charge signal is subsequently digitized 
2 5 to provide pixel data for the associated pixel corresponding 
to the pixel charge signal . 

Image sensor 10 0 can be used in black-and-white imaging 
applications or it can be used in color imaging 
applications. Besides digital cameras, image sensor 100 can 
30 also be applied in scanners, photocopy machines or facsimile 
machines to capture an image of an object by scanning the 
object sequentially. 
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In the case of a black-and-while application or a 
monochrome application, image sensor 100 is an array of 
photosensitive devices without the color filters. Image 
sensor 100 generates an image comprising an array of pixel 
5 data, each pixel data being a numerical value representing 
the intensity of the incident light impinged upon a pixel in 
the image sensor. For example, the pixel data can have a 
value between 0 and 255 for representing the gray scale or 
the variations of the light intensity of the scene being 

10 captured. 

For color applications, a mosaic of selectively 
transmissive filters is superimposed in registration with 
each of the photodetectors so that a first, second, and 
third selective group of photodetectors are made to sense 

15 three different color ranges, for example, the red, green, 
and blue range of the visible spectrum, respectively. As 
shown in Figure 1, image sensor 100 includes red, green and 
blue filters for capturing the red, green and blue 
components of a scene. A group of red, green and blue 

2 0 pixels, such as a pixel 106 for capturing the red spectrum, 

a pixel 107 for capturing the green spectrum, and a pixel 
108 for capturing the blue spectrum, is used to make up or 
compose a color pixel C(i, j) in a color image. 
Alternately, a color imaging system may include a beam 
25 splitter and separate image sensors, each disposed to 

separately capture a primary color component of the target 
image. A color pixel C(i, j) in such a color imaging system 
includes the corresponding or derived pixels in each of the 
three primary color images. 

3 0 Each color pixel C(i, j) in a color image is a vector 

pixel that may be expressed as follows: 
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C(i, j) = [R(i, j) G(i, j) B(i, j)] T 

where (i, j) are coordinates of an image pixel in the image 
sensor and C refers to the color image or images captured 
5 and R, G and B represent the intensity values for each of 
the three color spectra. If a cluster S of corresponding 
pixels in the color images have an identical value, namely 
R(i, j) =G(i, j) =B(i, j), where (i, j) is within S, a 
spot in the target (a scene or an object) corresponding to 
10 the cluster S is colorless, i.e. the spot is visually 

somewhere from black to white. Conversely if the cluster S 
of the pixels in color images have different values, i.e. 

R(i, j) ^ G(i, j) 9* B(i, j), the spot in the target 
corresponding to the cluster S is visually colorful. For 

15 example, a pure red, green or blue vector pixel are 

expressed as C (i, j) = [255 0 0] T , C (i, j) = [0 255 
0] T , or C (i, j) = [0 0 255] T , respectively. To ensure 
that a target scene or object can be exactly reproduced in a 
color image, the image sensor must be carefully controlled 

2 0 to produce color intensity values that can be combined to 
reproduce the colors of the target in the resultant color 
image . 

In general, image sensors have different responses to 
different wavelengths of light due to the absorption 

25 properties of the underlying photodetectors . For instance, 
silicon sensors are more sensitive to red light than blue 
light. Figure 3 illustrates response curves of an exemplary 
image sensor to the red, green and blue spectra. Red light 
has a shorter wavelength than blue light and can be more 

30 easily absorbed in most silicon sensors. The different 

response or sensitivities to the different light spectra in 
an image sensor can lead to undesirable image artifacts. 
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Figure 4 illustrates the intensity curves for a red, 
green, and blue pixel captured by image sensor 10 0 at a 
predetermined exposure time. When the integrated light 
intensities of the three color pixels are captured at the 
5 same exposure time, undesirable result may occur due to the 
different sensitivity of the photodetectors to the different 
color spectra. For example, as illustrated in Figure 4, 
because a silicon image sensor is more sensitive to red 
light than green or blue light, the red pixel may become 

10 saturated by a certain exposure time while the blue pixels 
have not yet integrated enough light at that same exposure 
time. That is, the photodetectors collecting red light may 
become over-exposed while those photodetectors collecting 
blue light may become underexposed. Previous techniques for 

15 compensating for the disparity in sensitivity levels involve 
stopping the light integration process when the image sensor 
detects that the red pixels have became saturated and then 
artificially saturating the blue and green color pixels. A 
color image thus captured may become distorted in color as 

20 the proportional ratio of each color components becomes 
misrepresented . 

What is needed is an image sensor with improved color 
reproduction capability so that color images can be rendered 
accurately. 

25 

SUMMARY OF THE INVENTION 

According to one embodiment of the present invention, 
an image sensor includes a sensor array including a two- 
dimensional array of pixel elements. The sensor array 
3 0 outputs digital signals as pixel data representing an image 
of a scene. The array of pixel elements includes a first 
group of photodetectors having a first sensitivity level and 
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a second group of photodetectors having a second sensitivity 
level. In operation, the first group of photodetectors 
generates the output signals after a first exposure time and 
the second group of photodetectors generates the output 
5 signals after a second exposure time, where the first 
exposure time and the second exposure time are within a 
snapshot of the scene and the first exposure time is 
different than the second exposure time. 

In another embodiment, an image sensor employs multiple 

10 sampling to capture an image of a scene . A different set of 
exposure times are assigned to each group of photodetectors 
having different sensitivity levels. The exposure times in 
each set of exposure times are chosen so that the signal to 
noise ratio is optimized for all groups of photodetectors. 

15 When thus operated, the image sensor of the present 

invention provides improved color reproduction capability 
and improved signal to noise ratio, especially for the less 
sensitive photosensitive elements. 

The present invention is better understood upon 

20 consideration of the detailed description below and the 
accompanying drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates an image sensor or photosensitive 
25 chip which may be used in an image capturing device. 

Figure 2 illustrates an exemplary structure of a 
photodetector where a photodiode is modeled as a current 
source and a capacitor . 

Figure 3 illustrates response curves of an exemplary 
3 0 image sensor to the red, green and blue spectra. 
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Figure 4 illustrates the intensity curves for a red, 
green, and blue pixel captured by the image sensor of Figure 
1 at a predetermined exposure time. 

Figure 5 illustrates the digital pixel sensor 
5 architecture used in the M25 patent which may be used to 
practice the color imaging method according to one 
embodiment of the present invention. 

Figure 6 is a block diagram of a digital image sensor 
where an A/D converter is associated with four 
10 photodetectors in an image sensor array. 

Figure 7 shows a functional block diagram of an image 
sensor which may be used to practice the color imaging 
method according to one embodiment of the present invention. 

Figure 8 shows a functional block diagram of an image 
15 sensor which may be used to practice the color imaging 
method according to another embodiment of the present 
invention. 

Figure 9A illustrates four rows of exemplary memory 
cells in a data memory used to store image information 
20 according to one embodiment of the present invention. 

Figure 9B illustrates four rows of exemplary memory 
cells for storing image information according to another 
embodiment of the present invention. 

Figure 10 illustrates the pixel intensity values vs. 
25 exposure time for four representative pixels A, B, C and D 
as detected by the DPS array of the image sensor in Figure 
8 . 

Figure 11 illustrates three sets of exposure times as 
applied to three representative color pixels R, G and B for 
30 sensing the red, green and blue color spectra of the visible 
light according to one embodiment of the present invention. 
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In the present disclosure, like objects which appear in 
more than one figure are provided with like reference 
numerals . 

5 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

According to the principles of the present invention, a 
digital image sensor uses sensitivity-dependent exposure 
control whereby different groups of photodetectors with 
different sensitivity levels are subjected to different 

10 amounts of exposure time. An image thus captured has 

improved signal to noise ratio (SNR) . The sensitivity- 
dependent exposure control imaging method according to the 
present invention has particular applications in color 
applications where the image sensor includes photosensitive 

15 elements having different sensitivities to different color 
spectra. By using the sensitivity-dependent exposure 
control imaging method, a color image thus captured can have 
improved color accuracy and improved signal to noise ratio, 
especially for the less sensitive photosensitive elements. 

20 The sensitivity-dependent exposure control imaging 

method of the present invention can be applied to digital 
image sensors. In the preferred embodiment, a digital pixel 
sensor is used. In other embodiments, the sensitivity- 
dependent exposure control imaging method can be practiced 

25 in a camera including separate imager sensors for each 
primary color component . 

In one embodiment, the sensitivity-dependent exposure 
control imaging method is practiced in image sensor 100 of 
Figure 1. As described above, for capturing a color image, 

30 image sensor 100 can include a mosaic of selectively 

transmissive filters superimposed and in registration with 
each of the photodetectors. As shown in Figure 1, three 
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types of color filters are provided to sense three different 
color ranges. In the present example, the red-green-blue 
(RGB) color space is used. Thus, a red filter overlies 
pixel 106, a green filter overlies pixel 107 and a blue 
5 filter overlies pixel 108. When thus configured, image 
sensor 100 includes three groups of photodetectors having 
different sensitivity level. For instance, the group of 
photodetectors disposed to detect the red spectrum is more 
sensitive than the groups disposed to detect the green and 

10 blue spectra. Furthermore, for typical image sensors, the 

group of photodetectors disposed to detect the blue spectrum 
is the least sensitive. 

The sensitivity-dependent exposure control imaging 
method of the present invention can be applied to image 

15 sensor 100 for improving the color reproduction capability 
of the image sensor, despite the disparity in sensitivity 
levels among the photodetectors. In operation, image sensor 
10 0 captures an image using different exposure times for 
each group of photodetectors with different sensitivity 

2 0 levels and the different exposure times are within a 

predetermined time for a snapshot of a scene. As used 
herein, the terms "exposure time" or "exposure" are used to 
refer to the time it takes to produce the output signals. 
In accordance with the present invention, the exposure time 

25 for each of the groups of photodetectors is confined within 
a snapshot of a scene. For example, if the time for a 
snapshot of a scene is set up to 1/120 second, one exposure 
time is stopped at 1/240 to prevent signals of a first group 
of photodetectors from being saturated while another 

30 exposure time is stopped at 1/120 to ensure sufficient 
exposure of a second group of photodetectors . 
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In general, the stronger the sensitivity level, the 
shorter the exposure time. Thus, at a first exposure time, 
the integrated light intensity collected by the group of red 
pixels (such as pixel 106) is readout. Then, at a second 
5 exposure time, the integrated light intensity collected by 
the group of green pixels (such as pixel 107) is readout. 
Finally, at a third exposure time, the integrated light 
intensity collected by the group of blue pixels (such as 
pixel 108) which are the least sensitive pixels, is readout. 

10 The sensitivity-dependent exposure control can be 

implemented in hardware in the control circuitry of image 
sensor 100. The sensitivity-dependent exposure control can 
also be implemented in software whereby instructions from a 
processor can be provided to image sensor 10 0 to control the 

15 exposure time and readout process of the image sensor. 

In a preferred embodiment of the present invention, a 
digital pixel sensor (DPS) is used for practicing the color 
imaging method of the present invention. When a DPS sensor 
is used, pixel-level sensitivity-dependent exposure control 

20 can be applied to capture a color image with improved color 
accuracy and signal to noise ratio characteristics. 

A CMOS image sensor with pixel level analog-to-digital 
conversion is described in U.S. Patent No. 5,461,425 of B. 
Fowler et al . (the M25 patent), which reference is 

25 incorporated herein by reference in its entirety. Figure 5 
duplicates Figure 1 of the M25 patent and illustrates the 
digital pixel sensor architecture used in the M25 patent 
which may be used to practice the color imaging method 
according to one embodiment of the present invention. In 

3 0 Figure 5, image sensor 10 is referred to herein as a digital 
pixel sensor (DPS) which provides a digital output signal at 
each pixel element representing the light intensity detected 
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by that pixel element . The combination of a photodetector 
and an analog-to-digital (A/D) converter in an area image 
sensor helps enhance detection accuracy and reduce power 
consumption, and improves overall system performance. 
5 In the present description, a digital pixel sensor 

(DPS) array or a sensor array refers to a digital image 
sensor having an array of photodetectors where each 
photodetector produces a digital output signal. The pixels 
of a DPS array are sometimes referred to as a "sensor pixel" 

10 or a "sensor element" or a "digital pixel," which terms are 
used to indicate that each of the photodetectors of a DPS 
array includes an analog- to-digital conversion (ADC) 
circuit, and is distinguishable from a conventional 
photodetector which includes a photodetector and produces an 

15 analog signal. The digital output signals of a DPS array 

have advantages over the conventional analog signals in that 
the digital signals can be read out at a much higher speed. 
Of course, other schemes for implementing a pixel level A/D 
conversion in an area image sensor may also be used in the 

2 0 image sensor of the present invention. 

In the digital pixel sensor architecture shown in 
Figure 5, a dedicated ADC scheme is used. That is, each 
pixel element 15 in sensor array 12 includes an ADC circuit. 
The imaging method of the present invention can also be 
25 practiced in other DPS architectures, including a shared ADC 
scheme. In the shared ADC scheme, instead of providing a 
dedicated ADC circuit to each photodetector in a sensor 
array, an ADC circuit is shared among a group of neighboring 
photodetectors . Figure 6 is a block diagram of a digital 

3 0 image sensor where an A/D converter is associated with four 

photodetectors in an image sensor array. In digital image 
sensor 2 0 of Figure 6, sensor array 22 includes a two- 
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dimensional array of photodetectors 24 . Instead of 
providing a dedicated ADC circuit to each photodetector , an 
ADC circuit 2 6 is shared among a group of four neighboring 
photodetectors 24. Each of the ADC circuit 26 performs A/D 
5 conversion of the output voltage signal by multiplexing 

between the four neighboring photodetectors. Although the 
image capture time becomes four times longer in the shared- 
ADC architecture of Figure 6 than the dedicated-ADC 
architecture of Figure 5, the shared-ADC architecture of 

10 Figure 6 has the advantage of retaining the benefits of 

pixel level analog-to-digital conversion while using a much 
smaller circuit area, thus reducing manufacturing cost and 
improving yield. The shared-ADC architecture is also 
described in "A 640 x 512 CMOS Image Sensor with Ultrawide 

15 Dynamic Range Floating-Point Pixel-Level ADC," by David X. 
D. Yang et al . , IEEE Journal of Solid-State Circuits, Vol. 
34, No. 12, December 1999, p. 1821-1834, which reference is 
incorporated by reference in its entirety. 

In one embodiment of the present invention, the ADC 

2 0 circuit of each digital pixel or each group of digital pixel 

is implemented using the Mult i- Channel Bit Serial (MCBS) 
analog- to-digital conversion technique described in U.S. 
Patent No. 5,801,657 of Fowler et al . The MCBS ADC 
technique of the x 657 patent can significantly improve the 
25 overall system performance while minimizing the size of the 
ADC circuit. Furthermore, as described in the "657 patent, 
an MCBS ADC has many advantages applicable to image 
acquisition and more importantly, facilitates high-speed 
readout. U.S. Patent Application Serial No. 09/274,202 

3 0 provides alternative ADC mechanisms that can significantly 

improve the overall system performance while minimizing the 
size of the A/D converters. Finally, copending and commonly 
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assigned U.S. Patent Application Serial NO. 09/823,443 
describes a multiplexed multi-channel bit serial ADC which 
may be advantageously applied to a DPS image sensor 
implementing a shared ADC architecture. The aforementioned 
5 patent and patent applications are incorporated herein by 
reference in their entireties. 

Copending and commonly assigned U.S. patent application 
No. 09/567,638, entitled "Integrated Digital Pixel Sensor 
Having a Sensing Area and a Digital Memory Area" of David 

10 Yang et al . , describes an integrated DPS sensor with an on- 
chip memory for storing at least one frame of pixel data. 
The incorporation of an on-chip memory in a DPS sensor 
alleviates the data transmission bottleneck problem 
associated with the use of an off-chip memory for storage of 

15 the pixel data. In particular, the integration of a memory 
with a DPS sensor makes feasible the use of multiple 
sampling for improving the quality of the captured images. 
Multiple sampling is a technique capable of achieving a wide 
dynamic range without many of the disadvantages associated 

2 0 with other dynamic range enhancement techniques, such as 
degradation in signal-to-noise ratio and increased 
implementation complexity. The aforementioned patent 
application is incorporated herein by reference in its 
entirety. 

25 According to one embodiment of the present invention, 

the sensitivity-dependent exposure control imaging method of 
the present invention is implemented using the DPS 
architecture of Figure 5. In a DPS architecture, 
independent exposure control can be realized by controlling 

30 the readout process on a pixel-by-pixel basis. 

According to an alternate embodiment of the present 
invention, the sensitivity-dependent exposure control 
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imaging method of the present invention is implemented using 
the shared-ADC DPS architecture of Figure 6. In particular, 
the shared-ADC architecture can be applied advantageously in 
the imaging method of the present invention by arranging 
pixels with different sensitivity levels to share one ADC 
circuit. For example, the 4-to-l shared ADC architecture of 
Figure 6 can be used to sense the Cyan, Magenta, Yellow and 
Green (CMYG) color space. Thus, the four pixels 24 sharing 
one ADC circuit 26 can be disposed to sense a cyan color, a 
magenta color, a yellow color and a black color each. Each 
of cyan, magenta, yellow and black color pixels is read out 
at a different exposure time than the other color pixels . 
Thus, when the four color pixels share one ADC circuit, the 
multiplexing between the four color pixels can be 
implemented according to the exposure times of each color 
pixel. In this manner, the shared-ADC architecture can be 
advantageously applied to save circuit space while improving 
color reproduction capability. Of course, the shared-ADC 
architecture can also be used for capturing color images 
using the RGB color space. 

The parent application (U.S. patent application No. 
09/567,786, hereinafter "the x 786 application") describes a 
method for facilitating image multiple sampling using a 
time- indexed approach. In a preferred embodiment of the 
present invention, the sensitivity-dependent exposure 
control imaging method practices multiple sampling for 
improving the sensitivity range of the photodetectors while 
improving the color accuracy of the image . 

Figure 7 duplicates Figure 3 of the '786 patent 
application and shows a functional block diagram of an image 
sensor 300 which may be used to practice the color imaging 
method according to one embodiment of the present invention. 
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The operation of image sensor 3 00 using multiple sampling is 
described in detail in the A 786 patent application. Image 
sensor 3 00 includes a DPS sensor array 3 02 which has an N by 
M array of pixel elements. Sensor array 3 02 employs either 
5 the sensor architecture shown in Figure 5 or Figure 6 and 
incorporates pixel level analog- to-digital conversion. A 
sense amplifier and latch circuit 3 04 is coupled to sensor 
array 3 02 to facilitate the readout of digital signals from 
sensor array 302. The digital signals (also referred to as 

10 digital pixel data) are stored in digital pixel data memory 
310. To support multiple sampling, image sensor 300 also 
includes a threshold memory 3 06 and a time index memory 308 
coupled to sensor array 302. Threshold memory 306 stores 
information of each pixel indicating whether the light 

15 intensity value measured by each pixel in sensor array 3 02 
has passed a predetermined threshold level. In this 
example, the information is stored as a one -bit threshold 
indicator bit. The exposure time indicating when the light 
intensity measured by each pixel has passed the threshold 

20 level is stored in time index memory 308. In this example, 
the time index value is a two-bit value identifying each 
time exposure. As a result of this memory configuration, 
each pixel element in sensor array 3 02 can be individually 
time-stamped by threshold memory 306 and time index memory 

25 308 and stored in digital pixel data memory 310. 

Figure 8 duplicates Figure 5 of the A 838 patent 
application and shows a functional block diagram of an image 
sensor 400 which may be used to practice the color imaging 
method according to another embodiment of the present 

3 0 invention. The operation of image sensor 400 using multiple 
sampling is described in detail in the ^838 patent 
application. Furthermore, image sensor 4 00 may employ the 
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image information storage method for facilitating multiple 
sampling as described in the A 838 patent application. 
Figures 9A and 9B illustrate memory configuration which can 
be used in memory 410 for implementing the image information 
5 storage method in accordance with embodiments of the present 
invention . 

With the memory configuration outlined above and 
illustrated in Figure 8, image sensor 400 can implement 
multiple sampling to improve the quality of an image. In 

10 multiple sampling, each pixel element is exposed to a target 
image for two or more different exposure times in order to 
compensate for bright and dark portions of the image. 
Additionally, the information regarding the exposure time 
associated with each pixel and the integrated intensity for 

15 that pixel is stored in memory 410 for use in computing the 
simulated pixel intensity when needed. 

Figure 10 illustrates the pixel intensity values vs. 
exposure time for four representative pixels A, B, C and D 
as detected by DPS array 302. Figure 10 illustrates the 

20 multiple sampling operation where one set of exposure times 
(IT, 2T and 4T) is used for all the pixels in DPS array 302. 
In color applications, DPS array 302 may include a mosaic of 
red, green and blue color filters superimposed on the sensor 
array and in registration with each of the photodetectors . 

25 Therefore, in color applications, pixels A, B, C and D may 
represent pixels in different color groups of 
photodetectors . 

In the multiple sampling operation of Figure 10, pixel 
values are first read out at an exposure time IT and a 

30 multiple sampling logic circuit in image sensor 400 performs 
a threshold comparison operation on the pixel values. The 
threshold comparison operation can be implemented in a 
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variety of ways. In the present example, a 50% threshold 
level is used. Thus, at time IT, the multiple sampling 
logic circuit of image sensor 400 compares the pixel values 
readout from each pixel element of DPS array 3 02 and 
5 determines which of the pixel intensity values exceeds the 
5 0% threshold level. For example, in Figure 10, pixel A has 
an intensity value exceeding the 50% threshold level while 
pixels B to D have intensity values below the threshold 
level. Pixel data for pixel A is recorded in data memory 

10 410. The pixel data value and the time index value for 
pixel A will be used by image sensor 400 to derive the 
resultant intensity value of pixel A. The multiple sampling 
process continues with sensor readout being taken after 
exposure times of 2T and 4T where the time 4T is the last 

15 exposure time and represents one snapshot of the scene. 

Each time the pixel intensity value of a pixel exceeds the 
50% threshold level, the threshold indicator bit of the 
associated pixel is set and the associated time index for 
that pixel is stored in the time index memory 308. The 

2 0 measured digital pixel values are stored in pixel data 

memory 410. 

When pixels A, B, C and D represent pixels of different 
color groups (e.g., the RGB color space), pixel A may 
represent the intensity level of a red pixel while pixel D 
25 may represent the intensity level of a blue pixel. As 

described above, because a typical silicon image sensor has 
poor sensitivity to blue light, the blue pixels may need to 
be integrated for an exposure time much longer than the red 
pixels in order to acquire sufficient signals values to give 

3 0 a satisfactory signal to noise ratio. When all of the 

pixels are exposed to the same set of exposure times as in 
Figure 10, it is very difficult to optimize the signal to 



-18- 



M-9535 US 
7S3683 vl 

noise ratio for different color pixels with different 
sensitivity levels. 

In accordance with another embodiment of the present 
invention, the sensitivity-dependent exposure control 
5 imaging method can also be applied to image sensor 4 00 of 
Figure 8 employing multiple sampling to enhance the dynamic 
range of the image sensor. Thus, DPS sensor array 3 02 of 
Figure 8 may include groups of photodetectors having 
different sensitivity levels. For example, the groups of 

10 photodetectors can be disposed to detect different color 

spectrum of the visible light, such as the RGB color space. 
When multiple sampling is applied, a different set of 
exposure times are assigned to each group of photodetectors. 
The exposure times are selected according to the sensitivity 

15 level of that group of photodetectors. Thus, the multiple 
sampling operation is performed by using multiple sets of 
exposure times, each group of photodetectors is sampled by 
its associated set of exposure times. For example, if the 
group of photodetectors is disposed to sense a red color, 

20 the sensitivity level of the photodetectors is strong and 

the set of exposure times used may be shorter. On the other 
hand, if the group of photodetectors is disposed to sense a 
blue color, the sensitivity level of the photodetectors is 
weak and the set of exposure times used may be longer. 

25 Figure 11 illustrates three sets of exposure times as 

applied to three representative color pixels R, G and B for 
sensing the red, green and blue color spectra of the visible 
light according to one embodiment of the present invention. 
Figure 11 illustrates the use of multiple sets of exposure 

3 0 times for performing multiple sampling in an image sensor 
including groups of photodetectors with different 
sensitivity level. Referring to Figure 11, the red pixel R 
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is being sampled at exposure times 1TR and 2TR, the green 
pixel G is being sampled at exposure times 1TG, 2TG and 3TG, 
and the blue pixel B is being sampled at exposure times 1TB, 
2TB and 3TB. All of the exposure times are within a 
5 snapshot of a scene. The exposure times are chosen so that 
the signal to noise ratio is optimized for all groups of 
photodetectors . For instance, because the red pixels have 
stronger sensitivity level, the red pixel R can become 
saturated sooner than the green or blue pixels. Thus, the 

10 set of exposure times for the red pixels can be selected to 
be earlier in the snapshot period. 

On the other hand, because the blue pixels have weaker 
sensitivity level, the set of exposure times for the blue 
pixels can be selected to be later in the snapshot period so 

15 that the blue pixels can be given longer time to integrate 
the incident light intensity. In the example shown in 
Figure 11, the first exposure time for the blue pixel is 
chosen to be after the last exposure time for the red pixel . 
In this manner, the light intensity level for the red pixels 

2 0 can be captured before the red pixels become saturated due 

to overexposed while the light intensity level for the blue 
pixels can be exposed for a longer period of time to provide 
enough integration so that the resultant light intensity 
values have improved signal to noise ratio. 
25 Of course, Figure 11 is illustrative only and other 

configurations of exposure times are possible to achieve the 
desired color reproduction characteristics and the desired 
signal to noise ratio. Furthermore, the exposure times 
within a set of exposure times do not have to be linearly 

3 0 spaced. The exposure times can be non- linearly spaced to 

achieve the desired result. For example, for sensing the 
weak sensitivity elements, the exposure times in the 
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beginning can be more sparsely spaced than the exposure 
times at the end of the snapshot period. 

In another embodiment, the sensitivity-dependent 
exposure control imaging method of the present invention is 
5 applied to an imaging system, such as a high-end camera, 

employing multiple image sensors to collect different color 
components of an image separately. Sensitivity-dependent 
exposure control can be provided by controlling the exposure 
time of each image sensor independently to equalize the 

10 different sensitivity level of each image sensor as 
described above . 

Furthermore, the sensitivity-dependent exposure control 
imaging method of the present invention can be applied to 
image sensors having photodetectors with any types of 

15 sensitivity variations. In the above descriptions, the 

different sensitivity levels are derived from color filters 
(such as RGB filters or CMYG color filters) . However, the 
imaging method of the present invention can also be applied 
in an image sensor regardless of the cause of the 

20 sensitivity variations in the image sensor, as long as the 
sensitivity variations can be quantified. 

The above detailed descriptions are provided to 
illustrate specific embodiments of the present invention and 
are not intended to be limiting. Numerous modifications and 

25 variations within the scope of the present invention are 

possible. The present invention is defined by the appended 
claims . 
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